Behind every robust result is a robust method: Perspectives from a case study and publication process in hydrological modelling

Models are commonplace in hydrological and environmental sciences. In scientific contexts, models used to encapsulate our current understanding of processes test new propositions. applied predict future conditions hence a cornerstone the decision-making chain civil protection (e.g., real-time flood forecasting) management planning seasonal streamflow forecasts inform water allocation, ecological health assessment, so forth). The importance models—and public expectations them—are only likely increase, as science's attention turns formidable challenges XXI century. increased reliance on places corresponding expectation model reliability robustness. Environmental failures can be costly—from loss time PhD researcher who adopts published that out non-robust, potentially catastrophic consequences during natural disasters such floods droughts if incorrectly anticipate events. These considerations might lead an impartial observer expect see increasing emphasis design verification. Paradoxically, there is evidence opposite trend taking place. For example, modern publication practices sciences seem place ever presentation “results,” rather than “methods” achieve them. Some journals minimise their methods sections by presenting them reduced font size, or even relegate supplementary materials do not form part formal printed versions. motivation for these changes perhaps understandable: results interpretation usually seen more exciting aspects endeavours. But degree prioritisation interpretations over methodological foundation brings real risks longer term rigour credibility. particularly high applications increasingly dominated heavily abstracted mathematical models, which simulate variables far beyond those we measure (Cunge, 2003). Seemingly routine choices significantly affect conclusions case studies investigations—making it problematic practice skimp details enthusiastic rush towards next great discovery able “demonstrate.” Our commentary emphasises value three instruments arguably remain under-utilised modelling broader applications: (1) benchmarking with null-hypothesis model, (2) testing predictions space–time validation, (3) carrying controlled comparisons. A schematic representation shown Figure 1, representative context developing catchment scale rainfall-runoff model. play key role classic method, certainly modelling. establishing meaningful benchmark has been emphasised many hydrologists (Pappenberger et al., 2015; Seibert 2018). need validation tests was notably advocated Klemeš (1986) followed numerous discussions topic Andréassian 2009; Refsgaard, 1997); also earlier discussion Burges (1984). systematic comparisons hypothesis reviewed elaborated Clark al. (2011) (see Baker, 2017; Pfister & Kirchner, 2017). Yet, necessarily included what generally perceived “good practice” common protocol, adopted many/most studies, relies calibration-validation (Blöschl Sivapalan, 1995; Cunge, 2003; 1997), but does explicitly consider alternative models. Validation space, although considered important spatially distributed implemented routinely (Refsgaard, 1997). Research sciences, including climate modelling, mirrors experiences, carefully designed being stressed several recent publications Eyring 2019; Rood, 2019). So, multiple opinion papers notwithstanding, why modelling? reason twofold. First still insufficient appreciation guidance should constitute practice.” An appropriate protocol unique, may depend type its application. some cases calibration advisable Second, how much matter practice. With exceptions Refsgaard Hansen, 2010), previous have revolved mainly around theoretical aspects. We therefore refrain from restating “theoretical” instead offer practical example. story inspired study Luxembourgish surprisingly wide range dynamics (Fenicia 2016). This revisits presents Hypothetical Paper Submission Review Process. start choice illustrate progressively stringent techniques change conclusions. characters humble Authors, Reviewers 1–3, Editor. disclaimer that, this story, real, any semblance actual persons events purely coincidental. begins Authors submitting manuscript hydrology 300 km2 Attert Luxembourg. catchment, diversity hydrograph observed across locations beseeched question climatic landscape properties acted dominant controls generation. research approached perspective, seeking find structure parameter values reproduced—or commonly put, “explained”—the variability. reader will appreciate setting up challenging endeavour. At very least, one develop conceptual translate into computer code, gather format data, interface algorithm, spend non-negligible amount debug run entire machinery. did all this, when were necessary, they best “do things right way,” following recommendations literature. After quick description methods, paper focused results. As examples choices, using robust stepping scheme avoid numerical instabilities (Kavetski Clark, 2011), square-root transformation account error heteroscedasticity/skew (McInerney Sorooshian Dracup, 1980), employed global optimisation strategies (Duan 1992; Qin validated separate period Klemeš, 1986; To ascertain model's ability streamflow, had “best practices” evaluated domain, signature domain (Hrachowitz 2014; Kavetski particular, addition Nash-Sutcliffe (NS) efficiency series simulations at subcatchments—an imperfect evaluation metric (Schaefli Gupta, 2007), familiar widely one—the signatures: runoff coefficient, baseflow index, flashiness index Addor 2018; Baker 2004). selection data performance metrics argued provide suitably comprehensive evaluation. proposed referred M1 (and numbering already foreshadows final one…). Like based concept Hydrological Response Units (HRUs). case, 3 HRUs defined, topography. Following standard practice, same “generic” assigned each HRU. main selling point capture spatial variability area. 2a terms NS subcatchments. discussing figure, noted “satisfactory” “acceptable” subcatchments one, efficiencies between 0.60 0.93. rogue subcatchment Huewelerbach headwater where 0.34. contended “minor” problem “idiosyncrasies” catchments (McDonnell, signatures, absolute difficult match, dots pattern, linear correlation (Pearson correlation, R) 0.77–0.94 (Figure 2b–d). Moreover, resulted moderate decrease performance. Overall, “credible” indication Satisfied findings, submitted work publication. They anticipating odyssey ahead. Then came reviews. 1 2 positive. general remarks improving referencing presentation. Reviewer 3, instead, critical. statements “acceptable”—these subjective, lacked frame reference—“with respect satisfactory?” reviewer requires credible benchmark, is, “null-hypothesis” 1a). argued, “Getting above zero ain't exactly newsworthy.” Setting trivial, literature examples. reflection, solution. most relevant setup HRUs, decision associates distinct elements. requested serve hypothesis, could represented rainfall inputs uniformly parameters setup, difference here M0, HRUs: single HRU M0 versus M1. made other decisions. revised 3. outperformed 9 10 subcatchments, reassuring result. Still, failed similarly Huewelerbach, once again attributed “idiosyncrasies.” clearly indices, R 0.94 0.09 respectively index. Notably, inclusion allowed replace subjective (“good” “bad”) relative ones (“better than” “worse than”). That said, diminished merits match coefficient better just virtue distributing forcings, without parameters. exception overturn conclusion topography offered “realistic” catchment. re-submitted paper, convinced thorough revision would satisfy critical reviewer… Authors' optimism lasted long re-review process. Actually, first two reviewers now about recognised improvements themselves asked for. additional set criticisms. particular concern “temporal split sample” sufficient asserted “a make time, space.” held back tenacious reviewer. However request eminently sensible hard dismiss. Editor perform last check, “should take work.” confident decided meet evade challenge, so-called “proxy-basins, split-sample” (Klemeš, 1986). calibrated group given period, another 1b). Horror damnation! Model performed considerably worse 4. (panel a), continued fail infamous More regrettably, pride joy M1—its simulated signatures—was lost, dropping 3), 0.46–0.53 validation. contemplated defending after experiment began it. Instead arguing hitherto favoured provided improvement simulating 6 catchments), embarked arduous journey revisiting assumptions. attempts, requests extend re-submission deadline … Eureka! M2, found reproduced M1! M2 well Huewelerbach—no blame idiosyncrasies no more! signatures (panels b–d), matched exceptionally indexes (R 0.89 0.97 respectively, whereas close zero). M1? Whereas assumed driven related geology control 4 HRUs. contrasting assumptions schematised 5a,b respectively. Despite larger number fewer (11 21, respectively). reduction obtained tailoring structures specific focusing processes, linking different original submission, change. outcome somewhat embarrassing nonetheless rewrote big chunk work, hoping satisfied. When received reviews, understood good sign, expecting short acceptance email. hide discomfort sending again. understandably, consult reviewers. So scroll down comments moralising preamble message changed through review process (as result adopting approaches), launched attack. It read: “Results show that? differ respects—not discretisation approach, represent HRU, presence links Are both differences pertinent? And impact? words, le raison d'être improved performance?” stoic adversity, faced concern. Two “intermediate” designed. M1A, solely namely define second M1B, regularisation, resulting comparison sequence illustrated 5c. Ultimately, enabled attribute predictive decisions 1c). summary, M1A showed largest variations, indicating decision. M1B lending credence idea achieved similar regularisation shaved off unnecessary complexity. overall (runoff coefficient) (baseflow indices). strengths, Revision paper. Their perseverance finally rewarded accepted. interested complete Fenicia (2016), correspond M-Uni, M-Top M-Geo-3, intermediate M-Geo-1 M-Geo-2. Note minor analysis (2016). (following recommendation McInerney (2017)). Luckily conclusions! straightforward apply, yet sobering impact While were, doubt, times frustrated exigence ultimately appreciated demonstrably higher quality robustness analyses. operations become approaches, complex abstract, correspondingly stringent. ongoing prioritising goes against need, foster laissez faire attitude, misused inexhaustible resources (non-robust misleading) findings recommendations. Detailed careful reviews protection, accomplished However, own, harsh often seems, cannot sole guardian weaknesses. Indeed weak, growing rate (perhaps inevitably) mirrored decline quality, limited assessment detailed scrutiny (Politi 2021). Relegating smaller giving less prominence presentations, exacerbate concerns. stronger awareness protocols—and keeping forefront reporting—is needed healthier attitude reduce risk non-robust findings. illustrates known protocols insightful argue explanatory and/or tools, “better” proposed, use investigated, adapted applications. Studies articulate targeted question, construct informative elucidate reasons straightforward, therein lies science art hope stimulate help implement work. thank Jim Buttle Michael Leonard constructive helped us improve manuscript.